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OTHER INPUTS OPTIONAL 
SUCH AS VOICE 



MAPPING ONE RGB COMPONENT OF A COLOR 
HERE, THE RED COMPONENT IS USED. THE SAME 
PROCESS CAN BE USED FOR THE GREEN AND BLUE 
COMPONENTS AS WELL. 

Ar - IS THE RED COMPONENT OF THE AQUA COLOR 

Or - IS THE RED COMPONENT OF THE ORANGE COLOR 

Pr - IS THE RED COMPONENT OF THE PIXEL COLOR 

Cr - IS THE RED COMPONENT OF THE COLOR ADJUSTED 
TO BE BETWEEN A AND 0 
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Cr = 0 





YES 







Cr = 1 







Cr = (Pr-Ar)/(Or-Ar) 
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YES 









Cr = 0 



Cr = (Pr-Ar)/(Or-Ar) 
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FIG. 6 



STEP 1 y_ 

CAPTURE PIXEL FIELD OF EVERY CAMERA FOR NEW VIDEO FRAME. DISTINGUISH EVERY TARGET IN EACH 
CAMERA FIELD FROM THE REST OF THE IMAGE. STORE THE PIXELS THAT MAKE UP EACH TARGET. 



STEP 2 



CAPTURE "INTERRUPT MEMBER" POSITIONS 



STEP 3 



DETERMINE THE ID OF THE WORK TOOL FROM THE UNIQUE TARGET AND SUBTARGET, COLOR, SHAPE, AND 
ORIENTATION INFORMATION. THIS IS DONE BY FINDING THE BEST MATCH OF THIS TARGET INFORMATION TO BE 
STORED IN A TABLE. EACH PATTERN STORED IN THE TABLE HAS A CORRESPONDING ID NUMBER. 



STEP 4 



DETERMINE X.Y.Z, AND ORIENTATION ANGLES A1. A2, A3 OF TARGETS USING PHOTOGRAPHIC OR LOOKUP TABLE 
PATTERN MATCHING TECHNIQUES. 



STEP 5 



COMPUTE THE LOCATION AND ORIENTATION OF THE WORK TOOL'S "ACTION TIP" BY APPLYING AN OFFSET VECTOR 
TO THE TARGET POSITION AND ANOTHER VECTOR CALCULATION IS USED TO MODIFY THE ORIENTATION. THIS 
INFORMATION IS ALSO CONVERTED TO THE PROPER APPLICATION COORDINATES. STORE THIS INFORMATION. 



STEP f 



CHECK TO SEE IF ANY INTERRUPT BUTTON HAS BEEN CHANGED. IF NO, REPEAT FIRST STEPS, ELSE GO ON. 
STEP 7A | ~~ 



DISTINGUISH UNIQUE COMBINATIONS OF INTERRUPT BUTTONS AND TARGET PATH INFORMATION (LOCATION, 
ORIENTATION AND TIMING PATTERNS) 

1 . ONE UNIQUE BUTTON/PATH COMBINATION DETERMINES PATH LOCATION, ORIENTATION AND TIMING 
INFORMATION (REFERRED TO AS PATHDATA). 

2. ALL OTHER COMBINATIONS IDENTIFY INPUT PARAMETERS OR OBJECTS OR FUNCTIONS OR PROGRAM 
INSTRUCTIONS (REFERRED TO AS OBJECTDATA). 



STEP 7B £ 



IF THIS IS PATHDATA, CONVERT IT TO THE APPROPRIATE TASK PATH INFORMATION AND USE THE EXTRA 
TARGET INFORMATION BEYOND THAT NEEDED FOR THE PATH TO DEFINE CONTROL PARAMETERS. 



STEP 7C 



IF THIS IS OBJECTDATA, FIND THE BEST MATCH OF A TARGET PATH OR SET OF TARGET PATHS INCLUDING 
THE SEVEN DIMENSIONAL DATA (LOCATION, ORIENTATION AND TIME) OF EACH TARGET AND THEIR 
CORRESPONDING WORK TOOL IDS. ASSOCIATE A UNIQUE NUMBER (REFER TO AS A QUANT) TO THE 
ABSOLUTE OR RELATIVE PATH INFORMATION. 



STEP 8 



INPUT THE QUANTS TO THE CONVERSION TABLE AND OUTPUT CORRESPONDING PROGRAM MENU SELECTIONS, 
MACRO INITIATIONS, OBJECT SELECTIONS, FUNCTION CALLS, PARAMETER SELECTIONS, ETC. IN OTHER 
WORDS, THE CONVERSION TABLE OUTPUT CAN BE USED TO DEFINE PROGRAM COMMANDS, OR REDEFINE THE 
OBJECT THAT IS ALIASED BY THE TARGETED TOOL OR ITS CORRESPONDING FUNCTIONS OR PARAMETERS. 
A USER DEFINED CONVERSION TABLE ALLOWS USERS FLEXIBILITY SUCH AS TO DEFINE THE PATH PATTERN THAI 
HE/SHE FINDS MOST COMFORTABLE. 



STEP 9 



APPLY CONVERSION TABLE OUTPUT TO A COMPUTER PROGRAM EITHER DIRECTLY OR VIA LINK SUCH AS INTERNET. 
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FIG. 8c 



PREREGISTER THE LOCATION AND ORIENTATION OF A 
SET OF OBJECTS WITH FREEFORM ATTACHMENTS 




ATTACH OBJECT TO OBJECT HOLDER 

1 zz 



ATTACH OBJECT HOLDER TO BASEBOARD 



ROTATE AND TRANSLATE OBJECT HOLDER 
TO DESIRED POSITION 



ATTACH APPROPRIATE TARGET TO OBJECT 
HOLDER SO THAT THE TARGET IS WITHIN THE 
TARGET VOLUME WITH AN ORIENTATION 
ROUGHLY FACING THE LENS PLANE OF THE 
CAMERA 



ENTER THE OBJECT ID, THE TARGET ID, AND 
THE ID OF THE ATTACHMENT LOCATION OF 
THE OBJECT HOLDER WHERE THE TARGET IS 
ATTACHED 



START THE CAMERA SYSTEM. ACQUIRE THE 
TARGET, COMPUTE THE ORIENTATION AND 
LOCATION OF THE OBJECT, UPDATE THE 
COMPUTER DATABASE 



I 



REMOVE TARGET 

3 



ADD TO A DATABASE DEFINING THIS OBJECT 
INCLUDING A) THE X.Y OF THE OBJECT CENTER. 
B) THE OBJECT PERIMETER PROJECTED ONTO 
THE BASEBOARD, C) THE GEOMETRIC 
DEFINITION OF THE OBJECT, D) MECHANICAL 
CHARACTERISTICS SUCH AS HOW A STEERING 
WHEEL WILL TURN, OR A SCALPEL WILL CUT 
INTO FLESH TILL IT HITS BONE, E) ASSOCIATED 
DATA SUCH AS QUICK LOOK UP OF DRUGS 
THAT CAN BE USED TO COMBAT A LEAKING 
COLON, F) ASSOCIATED COMPUTER PROGRAMS 
SUCH AS THOSE THAT COULD BE USED TO 
GUIDE A ROBOT CONTROLLED LAPROSCOPIC 
HEART SURGERY 



ATTACHED APPRORIATE TARGET TO BASEBOARD 
SO THAT THE TARGET IS WITHIN THE TARGET 
VOLUME WITH AN ORIENTATION ROUGHLY 
FACING THE LENS PLANE OF THE CAMERA 



ENTER THE BASEBOARD ID, THE TARGET ID, 
AND THE ID OF THE TARGET ATTACHMENT 
LOCATION ON THE BASEBOARD 



START THE CAMERA SYSTEM, ACQUIRE THE 
TARGET, COMPUTE THE ORIENTATION AND 
LOCATION OF THE BASEBOARD, UPDATE THE 
COMPUTER DATABASE 



REMOVE TARGET 



FIG. 8d 



PREREGISTER THE LOCATION AND ORIENTATION OF A 
SET OF OBJECTS WITH FIXED ATTACHMENTS 




ATTACH OBJECT TO OBJECT HOLDER THAT 
APPROPRIATELY ORIENTS AND RAISES OBJECT 



ATTACH OBJECT HOLDER TO BASEBOARD 
AT APPROPRIATE LOCATION 



ENTER THE OBJECT ID, THE OBJECT HOLDER 
ID, AND THE LOCATION AND ORIENTATION OF 
THE ATTACHMENT TO THE BASEBOARD 



I 



THE ASSOCIATED COMPUTER PROGRAM WILL 
ADD TO A DATABASE DEFINING THIS OBJECT 
INCLUDING A) THE X,Y OF THE OBJECT 
CENTER, B) THE OBJECT PERIMETER 
PROJECTED ONTO THE BASEBOARD, C) THE 
GEOMETRIC DEFINITION OF THE OBJECT, 
D) MECHANICAL CHARACTERISTICS SUCH AS 
HOW A STEERING WHEEL WILL TURN, OR A 
SCALPEL WILL CUT INTO FLESH TILL IT HITS 
BONE, E) ASSOCIATED DATA SUCH AS QUICK 
LOOK UP OF DRUGS THAT CAN BE USED TO 
COMBAT A LEAKING COLON, F) ASSOCIATED 
COMPUTER PROGRAMS SUCH AS THOSE THAT 
COULD BE USED TO GUIDE A ROBOT 
CONTROLLED LAPROSCOPIC HEART SURGERY. 



ATTACH APPRORIATE TARGET TO BASEBOARD 
SO THAT THE TARGET IS WITHIN THE TARGET 
VOLUME WITH AN ORIENTATION ROUGHLY 
FACING THE LENS PLANE OF THE CAMERA 



ENTER THE BASEBOARD ID, THE TARGET ID, 
AND THE ID OF THE TARGET ATTACHMENT 
LOCATION ON THE BASEBOARD 



START THE CAMERA SYSTEM, ACQUIRE THE 
TARGET, COMPUTE THE ORIENTATION AND 
LOCATION OF THE BASEBOARD. UPDATE THE 
COMPUTER DATABASE 



REMOVE TARGET 



FIG. 8e 



UTILIZING TARGETED POINTERS WITH A SET OF PREREGISTERED OBJECTS 

I 



CAPTURE TARGET ATTACHED TO POINTER. 



CAPTURE ANY INPUT FROM MEMBER. 



DETERMINE POINTER TIP LOCATION (X, Y, Z) 



DETERMINE NEAREST CENTER OF A PREREGISTERED 
OBJECT TO X,Y OF TIP. WE WILL CALL THIS OBJECT A 




CHECK Z OF POINTER AGAINST Z OF OBJECT'S 
UPPER SURFACE AT X,Y LOCATION 




APPLY MEMBER DATA AND OBJECT TO CURRENT COMPUTER 
PROGRAM WITH DATABASE, INFORMATION, FUNCTIONS, DATA, 
MECHANISMS DEPENDENT ON (X,Y,Z) OF POINTER TIP 




DATABASE DETERMINES IF FINGERTIPS' LOCATION SHOULD BE 
INTERPRETED AS POINTING OR GRIPPING, BASED ON THE 
OBJECT TYPE AND CHARACTERISTICS 
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